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Contemporary interest in STEM education is fueled, in part, by the 
poor performance of U.S. students on national and international assess¬ 
ments. According to a recent National Research Council (2011) report 
on STEM education in the United States, the National Assessment of 
Educational Progress (NAEP) indicates that approximately 75% of U.S. 
8th graders are not proficient in mathematics when they complete 8th 
grade, and, despite some progress over time, significant achievement 
gaps between racial and socioeconomic status subgroups persist in NAEP 
performance. Performance data also typically indicate lower levels of 
average performance in mathematics and science for U.S. students than 
for their counterparts in other countries with whom the United States 
competes in the global marketplace of commerce and information. For 
example, only 10% of U.S. 8th graders met the Trends in International 
Mathematics and Science Study advanced international benchmark in 
science, compared with 32% in Singapore and 25% in China (Gonzales 
et al., 2008). The central premise of this article is that at least one in¬ 
ternational assessment of mathematics and science, the Programme for 
International Student Assessment (PISA), is valuable for reasons that 
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go well beyond its use as a source of data; it sounds the alarm about 
low U.S. student achievement and, thus, motivates greater attention to 
STEM education. 

Triggered by the reported low levels of performance on domestic 
and international assessments of mathematics and science learning, 
much of the public policy conversation about the importance of STEM 
education focuses on the need to produce a cadre of well-prepared high 
school graduates. These students will, in turn, become the college gradu¬ 
ates who qualify for demanding jobs in STEM-related fields to keep the 
United States competitive in the global economy. Nevertheless, even if 
the STEM initiatives now underway in the United States to increase 
the number of students eligible to enter STEM-related careers are suc¬ 
cessful in their quest, the fact remains that the vast majority of precol¬ 
lege and college students in this country will not be employed in STEM 
fields when they complete their schooling. One influential report from 
the National Academy of Sciences, National Academy of Engineering, 
and Institute of Medicine (2011) eloquently captures this reality: 

The primary driver of the future economy and concomitant creation 
of jobs will be innovation, largely derived from advances in science 
and engineering.... 4 percent of the nation’s workforce is composed of 
scientists and engineers; this group disproportionately creates jobs for 
the other 96 percent, (p. 4) 

Given this reality, why press for a broad-based approach to enhanced 
student learning in STEM fields? One reason is that recognition of the 
importance of literacy in mathematics and science predates the identi¬ 
fication and naming of STEM as an integrated concept. For example, a 
focus on quantitative literacy has been advocated for the mathematics 
domain for over a decade (e.g., Madison & Steen, 2003; Steen, 2001), 
and Feinstein, Allen, and Jenkins (2013) make a similar argument for 
science: “Schools should help students access and interpret the science 
they need in response to specific practical problems, judge the credibility 
of scientific claims based on both evidence and institutional cues, and 
cultivate deep amateur involvement in science” (p. 314). An integrated 
exposure to STEM fields can cultivate in students, even those who do 
not pursue STEM-intensive careers, curiosity and passion that support 
a lifelong learning trajectory. 

Students also will benefit from STEM literacy in other facets of 
their lives. As an NRC report on STEM education noted, “Individual and 
societal decisions increasingly require some understanding of STEM, 
from comprehending medical diagnoses to evaluating competing claims 
about the environment to managing daily activities with a wide variety 
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of computer-based applications” (National Research Council, 2011, p. 3). 
High school and college graduates alike are called upon to participate in 
the democratic political process that is the hallmark of American citizen¬ 
ship. In that role, they participate in public debates and decisions about 
such issues as water quality and access, resource renewal in the face 
of global warming, and health care costs and quality. They also vote in 
elections that affect many STEM-related public policy issues, including 
those that influence local school boards and the U.S. Congress thereby 
shaping local and national investment in STEM education, research, 
and development. Additionally, they are likely to be the parents of many 
of the succeeding generations of potential contributors to the STEM 
enterprise. Thus, we think that it is important to consider all students, 
not only the mathematically and scientifically talented, as intended 
beneficiaries of increased attention to STEM education. 

This “STEM for All” perspective makes clear the critical need to en¬ 
sure that all students in our schools and colleges become STEM literate. 
From this perspective, we believe that PISA is an especially valuable 
resource for mathematics and science educators. If used appropriately, 
PISA can help promote the professional learning of U.S. teachers, thereby 
improving STEM teaching and learning in this country. 

PISA’s focus on literacy—the ability to use and apply knowledge and 
skills to real-world situations encountered in adult life—in mathemat¬ 
ics and science is aligned with STEM literacy and thus appears to be 
exactly the right emphasis to achieve the “STEM for All” goal. In this 
article, we focus primarily on the STEM discipline of mathematics, but we 
think that our proposal that PISA can be a valuable tool to support this 
endeavor also could be applied to the science and engineering domains 
of STEM, and we invite interested readers to consider that possibility. 

The PISA Assessment 

PISA is an international assessment of 15-year-old students’ read¬ 
ing, mathematics, and science literacy. PISA also includes measures 
of general or cross-curricular competencies, such as problem solving. 
PISA emphasizes functional skills that students have acquired as they 
near the end of compulsory schooling, as opposed to curriculum-based 
knowledge and skills. 

PISA is coordinated by the Organization for Economic Cooperation 
and Development (OECD), an intergovernmental organization of indus¬ 
trialized countries, and is conducted in the United States by the National 
Center for Education Statistics. PISA was first administered in 2000 and is 
conducted every three years. The most recent assessment was in 2012. 
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PISA Frameworks 

The frameworks guiding the PISA assessments reflect a consensus 
across OECD countries regarding the skills and abilities that demon¬ 
strate literacy in each content area. In 2012, mathematical literacy was 
defined follows: 

Mathematical literacy is an individual’s capacity to formulate, employ, 
and interpret mathematics in a variety of contexts. It includes reasoning 
mathematically and using mathematical concepts, procedures, facts and 
tools to describe, explain and predict phenomena. It assists individuals 
to recognize the role that mathematics plays in the world and to make 
the well-founded judgments and decisions needed by constructive, 
engaged and reflective citizens. (OECD, 2013, p. 25) 

The PISA 2012 mathematical literacy framework consisted of three 
main components: (1) mathematical processes and underlying funda¬ 
mental capabilities, (2) mathematical content, and (3) mathematical 
contexts (OECD, 2013). The process component encompasses much 
of what is commonly referred to as mathematical problem solving, 
including attention to (a) formulating situations mathematically; (b) 
employing mathematical concepts, facts, procedures, and reasoning; 
and (c) interpreting, applying, and evaluating mathematical outcomes. 
The PISA framework also identifies seven “fundamental mathematical 
capabilities” (e.g., communicating, representation, reasoning and argu¬ 
ment, using mathematical tools) that underlie the processes and that are 
akin to the mathematical practices in the Common Core State Standards 
(Common Core State Standards Initiative, 2012) and the mathematical 
process standards in Principles and Standards for School Mathematics 
(National Council of Teachers of Mathematics, 2000). 

The mathematics content assessed in PISA is defined by four over¬ 
arching ideas: quantity, space and shape, change and relationships, and 
uncertainty. Within each content area, specific mathematical topics are 
delineated. For example, the area of change and relationships encom¬ 
passes types of growth (e.g., linear, exponential, periodic, logistic) and 
relationships among types of growth; and the area of uncertainty includes 
probability, inference, and data collection/analysis/representation. 

PISA places heavy emphasis on authentic uses of mathematics and 
thus includes tasks that embed mathematics in contexts that might be 
encountered in real-world situations. Four types of situations or contexts 
are defined and used in developing the problems in PISA: personal, oc¬ 
cupational, societal, and scientific. 
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Key Features of PISA Tasks as Stimuli 
for Teacher Professional Learning 

PISA tends to be less well known in the U.S. education community 
than some other large-scale assessments, such as NAEP and Trends in 
International Study of Mathematics and Science (TIMSS), which have 
been with us for a longer time. Yet, several features of PISA make it a 
potentially useful resource for mathematics teacher education: 

1. PISA is relevant at both the middle school and secondary school levels. 
PISA is designed for and administered to a sample of 15-year-old students, 
an important point on the age/grade continuum, as content is likely to be 
highly relevant to teachers across the spectrum of grades 6-10. 

2. PISA treats commonly taught content in novel ways. PISA tests 
mathematical ideas that are likely to be at the core of the mathematics 
curriculum in grades 6-10. A comparison of the 2003 versions of NAEP, 
TIMSS, and PISA found that the mathematics topics addressed in all 
three tests were quite similar, but PISA is substantially different in 
how knowledge is assessed, especially with respect to applied problem 
solving (Neidorf, Binkley, Gattis & Nohara, 2006). Thus, the way that 
the mathematics content is represented in PISA tasks is likely to be 
atypical for mathematics teachers, especially secondary school teachers, 
making it likely that PISA tasks can serve well as stimuli for teacher 
inquiry and reflection. 

3. PISA focuses on quantitative literacy. PISA is intended to assess 
what some have called quantitative literacy (e.g., Madison & Steen, 2008), 
namely, students’ ability to use and apply mathematical knowledge and 
skills to authentic problem-solving situations or contexts encountered 
in adult life. Thus, the PISA tasks do not generally mirror standard 
mathematics textbook exercises, and teachers are likely to view the 
PISA tasks as embodying a novel form of mathematical proficiency. 

4. PISA tasks are cognitively complex. PISA’s focus on complex 
thinking and problem solving in real-world application contexts makes 
it a good source of intellectually rich tasks for mathematics teachers 
to consider. An examination of the cognitive complexity of mathemat¬ 
ics questions on three major assessments (PISA, NAEP, and TIMSS) 
found that PISA items were more likely than those on the other tests 
to be cognitively demanding (Neidorf et al., 2006). Another comparative 
analysis of PISA 2003 and TIMSS 2003 found that more than one-half 
of the PISA mathematics questions were classified as problem-solving 
tasks, the most cognitively complex category in the study, as opposed 
to less than one-third of the TIMSS questions (Dossey, McCrone, & 
O’Sullivan, 2006). 
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Illustrative PISA Tasks 

To illustrate these features, we include two publicly released PISA 
items, which are embedded in contexts that also may be of relevance to 
science educators. Both pertain to mathematics taught across a range of 
grades. The Power of the Wind (POW) task (Figure 1), for example, involves 
students’ computing basic calculations, solving an algebraic equation, 
and using the Pythagorean theorem. The juxtaposition of these topics 
in a single test item and the nonstandard question types found in the 
POW item also illustrate PISA’s atypical treatment of commonly taught 
mathematical content. The POW task requires quantitative literacy in 


Figure 1: Sample PISA Task: Power of the Wind 


POWER OF THE WIND 


Zedtown is considering building some wind power 
stations to produce electricity. 

The Zedtown Council gathered information about the 
following model. 



Model: 

Height of tower: 

Number of rotor blades: 
Length of one rotor blade: 
Maximum speed of rotation: 
Price for construction: 
Turnover: 

Maintenance cost: 
Efficiency: 


E-82 

138 metres 
3 

40 metres 

20 rotations per minute 
3 200 000 zeds 
0.10 zeds per kWh generated 
0.01 zeds per kWh generated 
Operational 97% of the year 


Note: kilowatt hours (kWh) is a measure of electrical energy. 


Question 1: POWER OF THE WIND 

Decide whether the following statements about the E-82 wind power station can be 
deduced from the information provided. Circle “Yes” or “No” for each statement. 


Statement 

Can this statement be deduced from 
the information provided? 

The construction of three of the power 
stations will cost more than 8 000 000 
zeds in total. 

Yes / No 

The maintenance costs for the power 
station correspond to approximately 5% 
of its turnover. 

Yes / No 

The maintenance costs for the wind 
power station depend on the amount of 
kWh generated. 

Yes / No 

On exactly 97 days a year, the wind 
power station is not operational. 

Yes / No 
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Question 2: POWER OF THE WIND 


PM922Q02 


Zedtown wants to estimate the costs and the profit that would be created by 
constructing this wind power station. 

Zedtown’s mayor proposes the following formula for estimating the financial gain, F 
zeds, over a number of years, y, if they build the E-82 model. 

F = 400 000 y - 3 200 000 


Profit from the 


Costs of building 

yearly production of 


the wind power 

electricity 


station 


Based on the mayor's formula, what is the minimum number of years of operation 
required to cover the cost of construction of the wind power station? 

A. 6 years 

B. 8 years 

C. 10 years 

D. 12 years 


Question 3: POWER OF THE WIND 


PM922Q03-0 1 9 


Zedtown has decided to erect some E-82 wind 
power stations in a square field 
(length = breadth = 500 m). 

According to building regulations, the minimum 
distance between the towers of two wind power 
stations of this model has to be five times the 
length of a rotor blade. 

The town mayor has made a suggestion for how 
to arrange the wind power stations in the Field. 
This is shown in the diagram opposite. 

Explain why the town mayor’s suggestion does 
not meet the building regulations. Support your 
arguments with calculations. 


250 m 



• = wind power station tower 
Note: Drawing is not to scale. 


Question 4: POWER OF THE WIND pm 922 qo 4 -o i 2 9 

What is the maximum speed that the ends of the rotor blades for the wind power 
station move? Describe your solution process and give the result in kilometres per 
hour (km/h). Refer back to the information about the E-82 model. 


Maximum speed: 


km/h 
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the form reading and interpreting numerical information, especially in 
question 1. 

The Decreasing C0 2 Levels (DCL) task (Figure 2) also poses a 
nonstandard task: Instead of asking students to calculate a percentage 
change, they are asked to demonstrate how a particular change was 
obtained. This item illustrates PISA’s focus on quantitative literacy, as it 
presents a unique and contextualized graph to read and interpret. This 
graph is an atypical blend of standard graphs that students typically 
learn to use in isolation from each other in mathematics classes. 


Figure 2: Sample PISA Task: Decreasing GO,, Levels 

M525: Decreasing CP 2 Levels 


Many scientists fear that the increasing level of C0 2 gas in our atmosphere is 
causing climate change. 

The diagram below shows the C0 2 emission levels in 1990 (the light bars) for several 
countries (or regions), the emission levels in 1998 (the dark bars), and the 
percentage change in emission levels between 1990 and 1998 (the arrows with 
percentages). 
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Typical of many PISA items, POW and DCL are cognitively complex 
tasks. Both involve multiple representations and several calculations, 
translation between problem context and mathematical models, reason¬ 
ing and interpretation, and precise communication skills (e.g., reading 
and interpreting quantitative information, expressing reasons and jus¬ 
tifications for claims, and evaluating one’s own answer or that proposed 
by another). For example, POW question 1 involves calculations, but 
students are not simply calculating. Rather, they are asked to draw on 
underlying mathematical knowledge to determine whether statements 
and calculations proposed by others are correct. Similarly, question 4 
requires a complex calculation that involves circumference, rates, and 
changing units. In DCL question 3, students are required to think about 
two different unstated possible ways to determine an increase and to 
find the maximum value using each method. 


Figure 2: Continued 


Question 1: DECREASING C0 2 LEVELS usisoouo 12 9 

In the diagram you can read that in the USA, the increase in C0 2 emission level from 
1990 to 1998 was 11%. 

Show the calculation to demonstrate how the 1 1 % is obtained. 


Question 2: DECREASING C0 2 LEVELS **525002-0 19 

Mandy analysed the diagram and claimed she discovered a mistake in the 
percentage change in emission levels: “The percentage decrease in Germany (16%) 
is bigger than the percentage decrease in the whole European Union (EU total, 4%). 
This is not possible, since Germany is part of the EU.” 

Do you agree with Mandy when she says this is not possible? Give an explanation to 
support your answer. 


Question 3: DECREASING C0 2 LEVELS 1*525003.0 1 2 9 

Mandy and Niels discussed which country (or region) had the largest increase of 
C0 2 emissions. 

Each came up with a different conclusion based on the diagram. 

Give two possible 'correct' answers to this question, and explain how you can obtain 
each of these answers. 
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Using PISA to Stimulate Teacher Professional Learning: 

The UPDATE Project 

Using PISA to Develop Activities for Teacher Education (UPDATE) 
is a project in which we have been exploring some potential uses of PISA 
tasks and data. We posit that PISA can be useful in much the same 
way as NAEP, which has long served as a key source of information 
for the U.S. mathematics education community. In UPDATE, we have 
developed some prototype, PISA-based materials and partnered with 
other professionals to use the materials in initial teacher preparation 
settings and teacher professional development contexts with teachers 
of mathematics in grades 6-11. 

Experience with NAEP suggests that the tasks used in complex, 
comprehensive mathematics assessments can be useful to educators. 
Some NAEP-based articles have focused on individual assessment tasks 
(e.g., Blume, Zawojewski, Silver, & Kenney, 1998; Kenney, Zawojewski, 
& Silver, 1998) or clusters of related tasks (e.g., Kenney & Silver, 1997; 
Stylianou, Kenney, Silver, & Alacaci, 2000) that afford an opportunity 
for teachers to “go deep” into issues tied directly to classroom instruc¬ 
tion and learning. Another approach has examined tasks and student 
performance with respect to a crosscutting issue of interest to teach¬ 
ers, such as equity (McGraw & D’Ambrosio, 2006; Silver, Strutchens, & 
Zawojewski, 1997; Strutchens & Silver, 2000) or mathematical problem 
solving and related processes (e.g., Silver, Alacaci, & Stylianou, 2000; 
Silver & Carpenter, 1989). 

To illustrate what might be possible with PISA tasks, we briefly 
describe the use of a PISA item, M136: Apples, successfully used in a 
mathematics professional development setting. The Apples task is one of 
50 publicly released PISA tasks after the 2006 assessment (http://www. 
oecd.org/pisa/38709418.pdD. 

Using the Apples Task in Teacher Professional Development 

The Apples task was used in the Developing Excellence in Learning 
and Teaching Algebra (DELTA) project, a mathematics teacher profes¬ 
sional development initiative in Michigan that focuses on building 
curricular coherence across grades 6-11 in the treatment of topics as¬ 
sociated with algebra. We partnered with the DELTA project to provide 
a context in which to test the materials that we were developing in the 
UPDATE project. Silver and Suh (in press) provide a detailed account 
of the DELTA project and the use of the Apples task in the project. In¬ 
terested readers can find more information there than we can provide 
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here, where we summarize some of the ways in which the Apples task 
was used to stimulate teacher professional learning. 

The DELTA variant of the Apples task (Figure 3) incorporated two 
modifications of the original PISA task. The phrase “pine tree” was used 
instead of conifer, and the wording of question 3.2 was modified. In the 
original PISA Apples task, the wording of question 3.2 reads as follows: 

There are two formulae you can use to calculate the number of apple trees 
and the number of conifer trees for the problem described above: 

Number of apple trees = n 2 

Number of conifer trees = 8 n 

Where n is the number of rows of apple trees. 

There is a value of n for which the number of apple trees equals the 
number of conifer trees. Find the value of n and show your method of 
calculating this. 


Figure 3: The Modified Apples Task Used in the DELTA Project 


Adapted from Mathematics Sample Tasks 
OECD's 2 009 PISA Assessment 

Mathematics Unit 3: Apples 

A farmer plants apple tree in a square pattern. In order to protect the apple trees 
against the wind he plants pine trees all around the orchard. 

Here you see a diagram of this situation where you can see the pattern of the 
apple trees and the pine trees for any number (n) of the rows of apple trees: 


X = pine 
? = apple tree 
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Figure 3: Continued 


Question 3.1 


Adapted from Mathematts Sample Tasks 
OECDs 2009 PISA Assessment 


Com 

n 

plete the table: 

Number of apple trees 

Number of pine trees 

1 

2 

1 

4 

8 

3 



4 



5 




Question 3.2 [Note: different wording than in original PISA task] 

Describe the pattern (words or symbols) so thatyou could find the number of apple trees 
for any stage in the pattern illustrated on the previous page: 


Describe the pattern (words or symbols) so thatyou could find the number of pine trees for 
any stage in the pattern illustrated on the previous page: 


For what value(s)of a will the number of apple trees equal the number of pine trees. Show 
your method of calculating this. 


Question 3.3 

Suppose the farmer wants to make a much larger orchard with many rows of trees. As the 
farmer makes the orchard bigger, which will increase more quickly: the number of apple 
trees or the number of pine trees? Explain how you found your answer. 
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The task modifications were intended to increase comprehension 
and accessibility for middle school students without affecting other key 
features of the task. In particular, the variation preserved the treat¬ 
ment of standard content in novel ways (e.g., juxtaposing a linear and 
quadratic pattern in the same problem context; including basic pattern 
finding with sophisticated reasoning about rates of change in the same 
item) and the cognitive complexity of the task (e.g., the use of multiple 
representations; calling for a range of processes, including analyzing, 
generalizing, and comparing). In fact, the modification actually may 
have increased the cognitive complexity of the task by making it more 
open-ended than was the original version. 

The modified Apples task was used in several different ways in the 
DELTA project on multiple occasions. We summarize here the varied uses 
of this single task in this one professional development initiative because 
we think that they are illustrative of a range of possible uses of many PISA 
tasks in teacher preparation and professional development settings. 

Teachers Solve the Problem 

When the Apples task was first presented to the DELTA teachers, 
they were asked to solve the problem individually. They met in small 
groups to discuss and compare solution approaches, which provided an 
opportunity for the teachers to familiarize themselves with the math¬ 
ematical concepts and skills associated with the problem. In this way, 
they were able to establish the relevance of the task to the mathematics 
that they teach, even though the task presentation likely differed from 
what they would find in the textbooks used in their classrooms. 

Teachers Predict How Students Will Solve the Problem 

After solving the problem, DELTA teachers were asked to antici¬ 
pate what students at the grade level they taught would be likely to 
do if asked to solve the problem. After working individually, they met 
in small grade-alike groups to develop a list of shared anticipations for 
students at each grade level. The expectations were recorded on posters 
and displayed for general discussion. 

The record of initial expectations came to play an important role in 
the learning of the DELTA project teachers. Subsequent examination of 
student work on the problem confirmed some of the teachers’ expectations 
and challenged others. As we saw in a subsequent session, the surprises 
afforded especially important opportunities for teacher learning. 

Teachers Examine the PISA Scoring Rubric 

Teachers were provided with the PISA scoring guide, which is available 
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for each of the publicly released tasks. They could see in the guide how 
PISA assigned points for various kinds of responses. Because the PISA 
task was modified when used in the DELTA project, only the portion of the 
rubric that pertained to the first and third questions was considered. 

Teachers Collect Student Work on the Problem 

As a homework assignment following the session in which they solved 
the Apples task, DELTA participants were asked to administer the task 
to at least one class of students, if feasible to do so. Collecting student 
work allows teachers to watch their own students solve the problem. It 
also provides a set of student responses that can be pooled across teach¬ 
ers to provide a more substantial sample of responses within and across 
grades. In DELTA, the teachers collected more than 900 responses from 
students in classrooms ranging from grades 5 through 12 and enrolled 
in a variety of mathematics courses (e.g., Algebra I, Algebra II, Pre-cal¬ 
culus). The diversity of student responses provided a rich resource for 
subsequent examination and analysis. 

Teachers Examine the Student Work on the Problem 

DELTA teachers were asked to examine the solutions produced by 
their students and then to meet with grade-level colleagues to examine 
all the student responses at their grade level. In their initial examination, 
they were asked to identify what the responses reveal about what students 
appear to understand and not to understand and what implications their 
observations might have for instruction. During the session, the grade-level 
group observations were recorded on poster paper and displayed in the 
room to facilitate a large group discussion that occurred later in the day. 

Just as students can sometimes make discoveries while exploring 
problem situations that influence their sense of identity and agency, this 
type of activity on the part of teachers—a minimally guided explora¬ 
tion of student work—may yield profound insights. Nevertheless, some 
teachers may benefit from a more structured approach. 

For a variety of reasons discussed by Silver and Suh (in press), 
including an emphasis on content coverage rather than on developing 
individual student understanding, many secondary school mathemat¬ 
ics and science teachers tend to focus almost exclusively on correctness 
when examining student work. Crespo (2000) and Davis (1997) charac¬ 
terize this as an evaluative rather than interpretive orientation toward 
teaching. A teacher using an evaluative orientation tends to listen to 
students’ ideas to judge them correct or incorrect and to diagnose and 
correct misunderstandings, similar to what Otero (2006) called a “get 
it or don’t” conception of formative assessment. 


Issues in Teacher Education 



Edward A. Silver & Rachel B. Snider 


25 


This orientation was quite apparent among many DELTA teachers 
when they initially examined the student work on the Apples task. The 
poster displays and the grade-level and whole group discussions focused 
almost exclusively on right/wrong categories and an elaboration of students’ 
errors and apparent misunderstandings, such as difficulties in setting up 
an equation to solve question 3.2c, missing 0 as a solution, rendering the 
repeated addition of 8 as n + 8 rather than as 8n, and confusing quadratic 
and exponential growth patterns. The professional development leaders 
had hoped for more attention to students’ understandings, so they decided 
that it would be beneficial to return to the student work one more time in 
a future session, with an eye toward shifting teachers’ attention to aspects 
of student performance other than correctness. 

Teachers Analytically Examine the Student Work on the Problem 

An UPDATE research team, including the authors of this paper, 
undertook an independent analysis of the student work on the modified 
Apples task, and paid particular attention to students’ use of representa¬ 
tions and strategies on questions 3.2 and 3.3. Two general observations 
emerged from our examination of the student work that we judged to 
have potential to engage the DELTA participants: 

• When making claims and representing generalizations, students 
in upper grades and advanced classes tended to use mathematical 
symbolism and equations, whereas, in middle school and in lower level 
mathematics classes, students relied more often on verbal descriptions. 

Yet, even in upper level classes, students often used verbal descriptions 
to express a generalization. 

• Some students at all grade levels used recursive strategies to solve 
subtasks 3.2a and 3.2b, with more using recursion for subtask 3.2b; 
students using recursion used only verbal descriptions rather than 
symbolic expressions to express their generalizations. 

This analysis suggested a scheme that might be useful in drawing 
teachers’ attention to more than right/wrong aspects of student work 
on the problem. Following our analysis of the student work, we created 
packets of student responses that contained specific examples to reflect 
the major strategies and representations evident in the full sample of 
student work: recursive description, recursive equation, explicit descrip¬ 
tion, and explicit equation. 

The response packets were used with the DELTA teachers in a 
subsequent professional development session, when the Apples task 
and student work once again became a focus of attention. Teachers were 
given the packets of student responses, and they were asked to sort the 
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responses to questions 3.2a and 3.2b into the following groupings: De¬ 
scribe recursive pattern in words; (Try to) Express a recursive pattern 
using symbolic notation; Describe an explicit pattern using words; and 
Express an explicit pattern using symbolic notation. 

Teachers Predict Frequency of Response Types 

DELTA teachers also were asked to predict the percentage of students 
who would be likely to produce each type of response at the grade level 
that they teach (i.e., 15% of grade 8 students will use words to describe 
a recursive pattern in question 3.2a). Teachers worked individually at 
first, then in pairs, and, finally, in grade level groups to compare and 
refine their predictions. 

Grade-level predictions were shared and discussed briefly in a 
whole-group session. In general, the predictions were that, as students 
progressed across the grades and through mathematics courses, they 
would become far more likely to express generalizations explicitly rather 
than recursively, and they would be far more likely to use symbolic ex¬ 
pressions and equations rather than verbal descriptions. Once again, 
by having the teachers make such predictions, the professional develop¬ 
ers hoped that the presentation of actual findings might include some 
surprises that could stimulate teacher learning. 

Teachers Consider a Comprehensive Analysis of Student Responses 

The UPDATE team presented its coding and analysis of the entire 
set of more than 900 student responses. For questions 3.2a and 3.2b, 
graphs were displayed to depict the frequency of student responses that 
expressed the generalization explicitly or recursively and that used verbal 
descriptions or symbolic expressions. The graphs vividly displayed the 
ways in which the student work aligned with or deviated from the teach¬ 
ers’ predictions. For example, when looking across the grades, the graphs 
revealed not only a trend toward expressing generalizations explicitly 
and with symbolic expressions but also an unexpected persistence of 
both recursive reasoning and verbal descriptions. 

The findings of the UPDATE analysis were discussed briefly in whole 
group, and then the participants met in grade-alike groups to discuss 
the findings and graphs as they pertained to their grade level. Teachers 
were encouraged to identify instructional issues raised by these find¬ 
ings—issues that pertained to their grade level and issues that might 
be pertinent across grade levels. Participants actively discussed and 
debated the findings and possible implications, moving fluidly between 
the graphs of general findings and the specific student responses that 
were available to them in the packets examined earlier in the day. Fol- 
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lowing discussion in grade-alike groups, the participants moved into 
cross-grade groups that mixed middle school and high school teachers. 
In these groups, participants discussed what the findings of this analysis 
suggested about what students were and were not learning from their 
mathematics instruction at each grade level and across grade levels 
as a means to increase curricular coherence, which was major point of 
emphasis in the DELTA project. 

Reflection on the Use of the Apples Task 

Although our presentation of the Apples task experience was neces¬ 
sarily brief and general, we think it embodies several points in regard to 
the use of PISA tasks as stimuli for STEM teacher professional learning. 
The first point is that the experience illustrates the diversity of ways 
that a PISA task might be used to stimulate teacher engagement and 
learning. The set of activity settings used in DELTA was extensive, and 
yet it represents only a sample of possibilities. Readers should be able 
both to generate other uses of the items for preservice and inservice 
teacher education settings and to think of variations on the specific 
activities and formats employed in DELTA. Moreover, it is important to 
think about the cumulative effects of a sequence of activities. In DELTA, 
the final professional learning activity appeared to have been critically 
important, but the experience of project participants in solving the tasks 
and predicting student solutions on prior occasions almost certainly 
played an important role in creating the learning opportunities that 
were manifested on that occasion. 

A second point is that PISA tasks can be used as found in PISA or 
modified to fit the needs of a particular teacher education context. The 
original PISA Apples task was a challenging mathematical task that 
treated important mathematics concepts and skills and allowed for many 
legitimate uses as a stimulus for teacher professional learning. Yet, the 
modification that was made when the task moved from PISA to DELTA 
turned out to be important for two reasons. First, although it retained the 
mathematical character of the original PISA task, it made the task more 
accessible to middle school students who had not yet been taught to write 
and solve algebraic equations. Second, the modification opened the door 
to students’ use of recursive reasoning to express the generalization. Our 
hunch is that recursion would have been far less likely to appear in the 
student work if the original PISA version of question 3.2 had been used, 
and the salience of recursion in the student work turned out to be a source 
of surprise for the teachers and, thus, an opportunity for their learning. 

A closely related point is that the mixing of middle school and high 
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school teachers in the participant group was useful for the teachers’ 
work with the Apples task. The hybridity of the participant group made 
available a range of perspectives on how students might solve the task, 
generated a rich sample of student work, and supported participants’ 
consideration of cross-grade curricular coherence issues. As Silver and 
Suh (in press) note, these factors played a role in the learning opportu¬ 
nities available to the DELTA teachers. 

A final point is the importance of designing activities in ways that 
allow teachers, especially secondary school teachers, to move beyond a 
simple right/wrong evaluation of student work. In DELTA, participants 
made significant progress when they were presented with specific ex¬ 
amples of student responses chosen in advance to represent particular 
strategies and representations and then directed to examine student 
responses using criteria that drew their attention toward matters of 
strategy and away from considerations of correctness. 

Coda 

If “STEM for All” is to succeed, we must ensure that all students in 
our schools and colleges become STEM literate. Teachers of mathemat¬ 
ics and science are central to the success of current initiatives aimed 
at more and better STEM education. Teachers will need both excellent 
preparation and strong support to do their part. As we have argued in 
this paper, PISA can be a valuable source of support for teachers, teacher 
educators, and professional development specialists in pursuit of the 
STEM agenda in the United States. The time has come to unleash the 
learning power of PISA! 
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